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ABSTRACT 

in this paper, 1 describe preliminaj^ work on a new research 
project in learning analytics at Arizona State University. In 
conjunction with an innovative remedial mathematics course 
using Khan Academy and student coaches, this study seeks to 
measure the effectiveness of visualized data in assisting student 
coaches as they help remedial math students achieve success in an 
online math class. 
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1. INTRODUCTION 

With 77,000 students, Arizona State University' has become one 
of the largest institution of higher learning in the United States 
second only to the University of Phoenix. A certain slice of newly 
enrolled students at Arizona State find themselves in a dilemma, 
they have been admitted to the university, but they have tailed to 
meet the minimum math requirement that would allow them to 
start taking undergraduate classes* 

These students are desirable for many reasons. They can help the 
university meet goals for diversity and social justice. Often these 
students are first generation college students. However, this 
population also poses challenges to the university'. Being unable 
to meet the minimum score on the mathematics placement test not 
only points to gaps in a student' s math background, it often points 
to larger issues of academic readiness. 

Overcoming the hurdle of meeting the minimum math 
requirement has been challenging. Online remedial math classes 
using an adaptive learning model have historically had a pass rate 
of around 50%. These pass rates have remained stubbornly low 
despite various efforts to improve them, 

2. A NEW APPROACH 

In the summer of 2014, EdPlus (ASU's online education arm) 
decided to launch a new version of this remedial math class built 
around Khan Academy and undergraduate peer coaches. One of 
the big reasons for making this change was data. Khan, because it 
is a non*profiL was open to sharing the data generated by students 
and KA’s strategies for student success. Arizona State wanted 
their remedial math classes to become their most data driven 
offering. Working with Khan had other advantages as well. 
Because KA has over 10 million unique users every month and 
over 2 billion problems worked, Khan is able to deploy and adapt 
its instruction at scale. 


3. CHALLENGES WITH KHAN 

While the strategy' of building a remedial math class around Khan 
Academy had many strengths, there were also significant 
challenges. The first challenge was in the form of a problem that 
many schools face when working with Khan Academy, While 
seeking to build a comprehensive universe of math instruction, 
KA has developed explicit pathways to math success. Khan calls 
these pathways “missions/' However, ASU's end goal: passing 
an exam that is meant to reflect the many math concepts that a 
student should know before entering college, cut across many 
Khan missions. In addition, Khan’s powcrftil analytical tools that 
are meant to aid instructors in following a student’s progress are 
tied to these missions. When math skills are being served up to 
students a la carte, as they are to accomplish ASU's remedial 
math program, the analytics are unraveled. 

A second major challenge facing the Khan Academy program was 
the same challenge facing the original remedial math program. 
Many of these students were failing to pass the minimum math 
requirement to enter Arizona State University because school in 
general has been chalknging for them for a very' long time. 
Putting these students by themselves in an online math course of 
any kind could be a recipe for failure. They needed additional 
support* The kind of personal attention these students need is very^ 
expensive* ASU decided to control that cost by employing a 
system of student “coaches.” Coming from a variety of majors and 
backgrounds, these student coaches were handpicked and given 
responsihilily for 20-25 online students each. Their job was to 
monitor, guide, tutor, and encourage these students to the end 
goal of having their coachees pass an exam that was meant to 
reflect their readiness to take college level math. 

In order to be effective, these coaches needed access to the data in 
Khan Academy about their students’ progress, but because ASU’s 
exam at the end of the remedial math course measured math skills 
that spanned several Khan math missions, the state-of-the-an 
Khan analytics that are lied to those missions were unavaUabie to 
the coaches. After a lot of work, (much of it spearheaded by the 
student coaches themselves) a spreadsheet was developed that was 
populated by weekly downloads of Khan data. It showed which 
math skills were practiced and which skills were mastered and 
matched these up to a rough metric that told the coaches whether 
their students were on track to successfully master all the math 
skills they needed before they had to take the exam* 

4. RESEACH GOALS 

The goal of my research is create custom data visualizations that 
fit ASU’s mission for this remedial math class and then measure 
the effectiveness of these analytics in assisting the student coaches 
in their work of creating student success. These analytics are 
specifically aimed at enabling the student coaches to visualize the 
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large amounts of data generated through Khan Academy. Khan 
stores data on each student*s attempt to solve a math problem 
related to a particular skill There is also data on how many times 
a student views a Khan video on a math concept or asks for hints 
when attempting a math problem. Because we are re-envisioning 
the analytics from the ground up, we have an opportunity to create 
analytics that are similar to the ones that Khan has created for its 
missions yet improve these analytics for ASU’s specific purposes 
and create dashboards that visualize the data in other ways that 
may be even more usefijl for the student coaches. Because the 
coaches only had access to Khan data through a spreadsheet the 
first semester the new remedial math class was taught, there is an 
opportunity to compare the success of coaches assisting their 
students with the spreadsheet data versus those using the more 
sophisticated data visualizations produced Ifom the student's 
actions within Khan Academy. 

in order to achieve this goal, ASU is teaming up with Blue 
Canary, a learning analytics company headquartered in Chandler, 
Arizona. Blue Canary and I are working directly with Khan 
Academy to address data flow issues including creating APPs that 
Will automatically access data from Khan databases that will be 
feeding the dashboards and graphics created for the student 
coaches. 1 am also going to be w^orking with Blue Canary to cr^te 
dashboards and data visualization tools for the Khan online math 
class in Tableau, These dashboards are directly aimed at assisting 
student coaches while they help their math coachees achieve 
success, 

5. RESEARCH QUESTIONS 

Once the dashboards are created and the student coaches start 
using them to assist their math students, we can start to address 
this research question: Do data visualization tools enable student 
coaches to better assist remedial math students entering Arizona 
State University achieve success? 


6. RESEARCH DESIGN 

The preliminajy design of this study is to compare data generated 
by two cohorts of remedial math students. The first cohort has 
been guided by student coaches who have been accessing the 
Khan data on through a spreadsheet created to keep track of skill 
practice and mastery. The second cohort will be guided by student 
coaches who have access to the data visualization tools and 
dashboards created by myself and Blue Canary in Tableau. The 
ultimate measure of coach success will be the pass rate of their 
students at the end of the course. In addition, there will be many 
other metrics to measure, as well, such as student engagement and 
persistence. 

This research is in the early stages, Mike Sharkey from Blue 
Canary and myself have been meeting with student coaches and 
instructional designers of the remedial math program to assess the 
needs of the student coaches and talk about possible data 
visualizations that may be helpful. API’s are being designed pull 
data from Khan for the analytics and dashboard layouts. While we 
are working on this, data is being generated by students in Khan 
Academy who are working with coaches that are relying on the 
spreadsheet to access data about the progress of their students in 
Khan. I am currently in second year of a four year PhD program, 
so we have some time to make adjustments and work out 
problems as they arise. 
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